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THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
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earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

!)□ Responsive to communication(s) filed on . 

2a)D This action is FINAL. 2b)K This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1 935 CD. 1 1 , 453 O.G, 21 3. 

Disposition of Claims 

4) E3 Claim(s) 1-19 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) K Claimfs) 1.2 and 4-19 is/are rejected. r 

7) K Claim(s) 3 is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) S The specification is objected to by the Examiner. 

1 0) El The drawing(s) filed on 01 June 2001 is/are: a)D accepted or b)H objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 
a)D All b)Q Some * c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2.Q Certified copies of the priority documents have been received in Application No. . 



3.Q Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
See the attached detailed Office action for a list of the certified copies not received. 



Attachment(s) 

1) K Notice of References Cited (PTO-892) 

2) [H Notice of Draftsperson's Patent Drawing Review (PTO-948) 

3) K Information Disclosure Statement(s) (PT0-1449 or PTO/SB/08) 

Paper No(s)/Mail Date 2. 



4) O Interview Summary (PTO-413) 

Paper No(s)/Mail Date. . 

5) O Notice of Informal Patent Application (PTO-152) 

6) □ Other: . 



U.S, Patent and Trademark Office 
PTOL-326 (Rev. 1-04) 



Office Action Summary 



Part of Paper No./Mail Date 4 



Application/Control Number: 09/803,802 Page 2 

Art Unit: 2621 

DETAILED ACTION 
Drawings 

1. The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because 
reference character "50" of Fig. 10 has been used to designate both the Y-axis on page 10, line 
26 and page 1 1 line 7 of the specification and the focal point on page 1 1, lines 2 and 4 of the 
specification. The drawings are also objected to as failing to comply with 37 CFR 1.84(p)(5) 
because they include the following reference sign(s) not mentioned in the description: "612" in 
Fig. 6 referring to the step of declaring a constant offset difference. A proposed drawing 
correction, corrected drawings, or amendment to the specification to add the reference sign(s) in 
the description, are required in reply to the Office action to avoid abandonment of the 
application. The objection to the drawings will not be held in abeyance. 

2. Figure 9 should be designated by a legend such as —Prior Art— because only that which is 
old is illustrated. See MPEP § 608.02(g). A proposed drawing correction or corrected drawings 
are required in reply to the Office action to avoid abandonment of the application. The objection 
to the drawings will not be held in abeyance. 

Specification 

3. The disclosure is objected to because of the following informalities: The examiner 
requests that the applicant update the status of the U.S. patent applications referenced throughout 
the specification. For instance, U.S. Patent Application No. 09/ 572,522 mentioned on page 2, 
line 1 8 of the specification. 

Appropriate correction is required. 
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Claim Objections 

4. The following quotation of 37 CFR § 1 .75(a) is the basis of objection: 

(a) The specification must conclude with a claim particularly pointing out and distinctly claiming the 
subject matter which the applicant regards as his invention or discovery. 

5. Claim 19 is objected to under 37 CFR § 1.75 as failing to particularly point out and 
distinctly claim the subject matter which the applicant regards as his invention or discovery. 

Claim 19, as currently presented, is dependent on claim 12. However, claim 12 is directed 
towards a method, while claim 19 is directed towards a computer program product. The 
examiner suggests amending claim 19 to correctly depend from claim 17 or 18. For examination 
purposes it will be assumed that claim 19 depends from claim 17. 

Claim Rejections - 35 USC § 102 

6. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by another filed 
in the United States before the invention by the applicant for patent or (2) a patent granted on an application for 
patent by another filed in the United States before the invention by the applicant for patent, except that an 
international application filed under the treaty defined in section 351(a) shall have the effects for purposes of this 
subsection of an application filed in the United States only if the international application designated the United 
States and was published under Article 21(2) of such treaty in the English language. 

7. Claims 1, 2, 4-9, and 14-19 are rejected under 35 U.S.C. 102(e) as being anticipated by 
Szeliski et al. (U.S. Patent No. 6,044,181 A). 

As applied to claim 1, Szeliski et al. discloses a method for deriving a three-dimensional 
panorama from a plurality of images of a scene generated by a range imaging camera of the type 
that produces ambiguities in range information, said method comprising the steps of: (a) 
acquiring a plurality of adjacent images of the scene, wherein there is an overlap region between 
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the adjacent images and at least some of the adjacent images are range images (see Fig. 3 and 
column 9, lines 37-41: The reference describes that a camera captures a sequence of images each 
having a focal length from the optical center (i.e. range images). As can be seen in Fig. 3 each of 
these images is adjacent to the next image and has an overlap region.); (b) providing offset data 
for the range images in order to recover corrected relative scene spatial information (see column 
6, lines 62-65: The reference describes that the image's transformation and focal length are 
modified (i.e. providing offset data) to minimize error (i.e. correct relative scene spatial 
information).), wherein the step of providing offset data further comprises: (i) detecting a relative 
range difference between adjacent range images as a constant offset between the adjacent images 
(see column 12, lines 11-14: The reference describes that an incremental deformation is 
computed (i.e. detecting a relative range difference between adjacent range images as a constant 
offset between the adjacent images) to improve registration between adjacent images.); (ii) 
applying the constant offset to at least one of adjacent range images to correct for ambiguities in 
the relative ranges of the range images, thereby providing corrected range images (see column 
12, lines 14-19: The reference describes transforming the image coordinates with a transform 
matrix that has been updated using the incremental deformation (i.e. constant offset), thus 
producing a warped image that has been corrected for ambiguities.); and (c) deriving a three- 
dimensional panorama from the corrected range images (see Fig. 3 and column 9, lines 48-49: 
As can be seen in Fig, 3, a three dimensional panorama is constructed from the images.). 

As applied to claim 2, Szeliski et al. discloses that the scene spatial information is 
provided as image values in a local three-dimensional coordinate system of each of the images 
(see column 9, lines 43-45: The reference describes that each of the images has a local 
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coordinate system.) and the step (c) of deriving a three-dimensional panorama comprises the 
steps of: (i) transforming the image values from each of the local three-dimensional coordinate 
systems of each of the images to a selected reference three-dimensional world coordinate system, 
thereby providing transformed range images (see Fig. 4: As can be seen in the Fig. 4, the image 
values are transformed from their local coordinate system onto a world 3D coordinate system 
(i.e. a selected reference three-dimensional world coordinate system).); (ii) warping the 
transformed range images onto a cylindrical surface, and forming a plurality of warped range 
images (see column 12, lines 1-4: The reference describes warping each image to a cylindrical 
coordinates images. Therefore a plurality of warped images are obtained.); (iii) registering 
adjacent warped range images (see column 9, lines 59-60: The reference describes that the 
purpose of the previously mentioned procedure is to register two images.); and (iv) deriving the 
three-dimensional panorama using the warped range images (see Fig. 3 and column 9, lines 48- 
49: As can be seen in Fig, 3, a three dimensional panorama is constructed from the images.). 

As applied to claim 4, Szeliski et al. discloses that a method for deriving a three- 
dimensional panorama from a plurality of images of a scene generated from a range imaging 
camera of the type that produces ambiguities in range information, said method comprising the 
steps of: (a) acquiring a plurality of images of the scene by rotating the camera about a Y-axis 
(vertical axis), wherein there is an inter-overlap region between adjacent images (see Fig. 3 and 
column 9, lines 34-37: The reference describes that the camera moves in a panning motion (i.e. 
rotated about a y-axis). As can be seen in Fig. 3, a plurality of images are acquired and there is 
an inter-overlap region between adjacent images.); (b) providing offset data for each image to 
recover corrected relative scene spatial information (X, Y, Z) with respect to a local XYZ 
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coordinate system (see column 6, lines 62-65: The reference describes that the image's 
transformation and focal length are modified (i.e. providing offset data) to minimize error (i.e. 
correct relative scene spatial information).); (c) selecting a reference three-dimensional world 
coordinate system against which spatial information of the scene can be correctly presented (see 
Fig. 4: As can be seen in the figure, a world 3D coordinate system is selected.); (d) transforming 
the collected relative scene spatial information (X, Y, Z) from each of the local three- 
dimensional coordinate systems of each of the images to the selected reference three-dimensional 
world coordinate system, thereby providing transformed (X, Y, Z) images (see column 10, lines 
7-55: The reference describes a planar perspective transformation that transforms the images to 
the selected three-dimensional world coordinate system.); (e) warping the transformed (X, Y, Z) 
images onto a cylindrical surface, and forming a plurality of warped (X, Y, Z) images (see 
column 12, lines 1-4: The reference describes warping each image to a cylindrical coordinates 
images. Therefore a plurality of warped images are obtained.); (f) registering adjacent warped 
(X, Y, Z) images (see column 9, lines 59-60: The reference describes that the purpose of the 
previously mentioned procedure is to register two images.); and (g) forming a three-dimensional 
(X, Y, Z) panorama using the warped (X, Y, Z) images (see Fig. 3 and column 9, lines 48-49: As 
can be seen in Fig, 3, a three dimensional panorama is constructed from the images.). 

As applied to claim 5, Szeliski et al. discloses that the plurality of images includes range 
images (column 9, lines 37-41: The reference describes that a camera captures a sequence of 
images each having a focal length from the optical center (i.e. range images).) and the step (b) of 
providing offset data further comprises the steps of: (i) detecting differences in constant offset 
between the range images (see column 12, lines 11-14: The reference describes that an 
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incremental deformation is computed (i.e. detecting differences in constant offset between the 
range images) to improve registration between adjacent images.); and (ii) using the differences to 
correct for ambiguities between the range images (see column 12, lines 14-19: The reference 
describes transforming the image coordinates with a transform matrix that has been updated 
using the incremental deformation, thus producing a warped image that has been corrected for 
ambiguities (i.e. using the differences to correct for ambiguities between the range images).). 

As applied to claim 6, Szeliski et al. discloses that the plurality of images generated from 
the range imaging camera includes color images and the three dimensional panorama is in color 
(see column 27, lines 20-23: The reference describes the use of color images.). 

As applied to claim 7, Szeliski et al. discloses that the reference three-dimensional world 
coordinate system is an arbitrary three-dimensional coordinate system (see Fig. 4: As can be seen 
from the figure, the world 3D coordinate system is an arbitrary three-dimensional coordinate 
system.). 

As applied to claim 8, Szeliski et al. discloses the step of selecting the reference three- 
dimensional world coordinate system from a three-dimensional coordinate system defined 
elsewhere (see Fig. 4: As can be seen from the figure, the world 3D coordinate system is defined 
elsewhere.). 

As applied to claim 9, Szeliski et al. discloses that transforming the corrected relative 
scene spatial information (X, Y, Z) comprises forming a homogeneous transformation matrix 
(see column 10, lines 1 1-21: Equation 2 calls for the use of a homogeneous transformation 
matrix.). 
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As applied to claims 14 and 15, which merely call for a system for performing the 
method of claims 1 and 4, Szeliski et al. discloses such a system as can be seen in Fig. 2B. 

As applied to claims 16-19, which merely call for a computer program product having a 
computer program for performing the method of claims 1, 4, 5, and 9, respectively, Szeliski et al. 
discloses such a computer program product since all of the processing performed by Szeliski et 
al. is performed by computer. 

Claim Rejections - 35 USC §103 

8. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

9. Claims 10-13 are rejected under 35 U.S.C. 103(a) as being obvious over the combination 
of Szeliski et al. (U.S. Patent No. 6,044,181 A) and Ray et al. (U.S. Patent No. 6,456,793 Bl). 
The arguments as to the relevance of Szeliski et al. in the rejection of claim 4 above are 
incorporated herein. 

The applied reference, Ray et al., has a common assignee with the instant application. 
Based upon the earlier effective U.S. filing date of the reference, it constitutes prior art only 
under 35 U.S.C. 102(e). This rejection under 35 U.S.C. 103(a) might be overcome by: (1) a 
showing under 37 CFR 1.132 that any invention disclosed but not claimed in the reference was 
derived from the inventor of this application and is thus not an invention "by another"; (2) a 
showing of a date of invention for the claimed subject matter of the application which 
corresponds to subject matter disclosed but not claimed in the reference, prior to the effective 



Application/Control Number: 09/803,802 Page 9 

Art Unit: 2621 

U.S. filing date of the reference under 37 CFR 1.131; or (3) an oath or declaration under 37 CFR 
1.130 stating that the application and reference are currently owned by the same party and that 
the inventor named in the application is the prior inventor under 35 U.S.C. 104, together with a 
terminal disclaimer in accordance with 37 CFR 1.321(c). For applications filed on or after 
November 29, 1999, this rejection might also be overcome by showing that the subject matter of 
the reference and the claimed invention were, at the time the invention was made, owned by the 
same person or subject to an obligation of assignment to the same person. See MPEP § 
706.02(1)(1) and § 706.02(1)(2). 

Szeliski et al. does not disclose that each image is captured as a bundle of associated 
images, nor does the reference disclose the particulars of such a bundle. 

Ray et al., in the same field of endeavor of image processing and the same problem 
solving area of range image formation discloses capturing an image in such a manner and the 
particulars of such image capture. 

In particular, as applied to claim 10, Ray et al. discloses that each image is captured as a 
bundle of associated images, said bundle including a plurality of phase images each 
incorporating the effect of a predetermined modulation frequency together with a phase offset 
unique for each image (see Fig, 3 and column 7, lines 38-65: The reference describes that each 
image is captured as a bundle of associated images and that the bundle includes at least three 
phase images each incorporating the effect of a predetermined modulating frequency and a phase 
offset.). 
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As applied to claim 1 1, Ray et al. discloses that each range image is generated from a 
respective plurality of phase images associated with each bundle (see column 7, line 40: The 
reference describes that the range image is generated from at least three phase images.). 

As applied to claim 13, which is representative of claim 12, Ray et al. discloses that the 
bundle also includes a color image (see column 7, lines 46-48: The reference describes that the 
bundle includes a range image.). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Szeliski et al. by capturing each image as a bundle of associated 
images as taught by Ray et al. because such a system allows for the "capture [of] ranging 
information without the sacrificing color information that would otherwise be available for 
capture" (see Ray et al.: column 4, lines 1 1-13). 

Therefore, it would have been obvious to combine Szeliski et al. with Ray et al, to obtain 
the invention as specified in claims 10-13. 

Allowable Subject Matter 

10. Claim 3 is objected to as being dependent upon a rejected base claim, but would be 
allowable if rewritten in independent form including all of the limitations of the base claim and 
any intervening claims. 

Conclusion 

11. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 




Application/Control Number: 09/803,802 



Page 1 1 



Art Unit: 2621 

Hsieh et al. (U.S. Patent No. 6,01 1,558 A) is pertinent in that the reference discloses 
stitching a pair of images together by warping the pair of images into an environment map 
representative of a panoramic image. 

Chen et al. (the article titled "A Practical Approach to Creating Environment Maps and 
Using Digital Images") is pertinent in that the article presents a system similar to the claimed 
invention. This reference is not prior art. 

12. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Ryan J. Miller whose telephone number is (703) 306-4142. The 
examiner can normally be reached on M-F 8:00-4:30. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Leo H. Boudreau can be reached on (703) 305-4706. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



Ryan J. Miller 
Examiner 
Art Unit 2621 
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